Slides for proposed talk at 
Hanyang University, Seoul, Korea 
March 31, 2005 ^tsntstivB^ 

"High-Fidelity Simulation in Biomedical and Aerospace Engineering” 

By Dochan Kwak 


Export Control Information 

All the material in this presentation are in public domain and have been disseminated previously in the 
following publications and meeting presentations 

• 3rd International Conference in CFD, Toronto, Canada, July 2004 
(plenary talk by Ron Bailey, also written version in the proceedings) 

• D. Kwak and C. Kiris, “Successes and Challenges of Incompressible Flow Simulation” presented at the 
16 th AIAA Computational Fluid dynamics Conference, Orlando, FL, June 23-26, 2003 

• Super Computing 2004 

» Kim, C. S., Kiris, C. and Kwak, D., 'Numerical Models of Human Circulatory System under Altered 
Gravity,’ AIAA Paper 2004-1092, AIAA 42nd Aerospace Sciences Meeting, Reno. Nevada, Jan. 5-8. 
2004. 

• Kwak, D., Kiris, C. and Kim, C.S., ‘Computational Challenges of Viscous Incompressible Flows,' 
Computers & Fluids 34 , pp283-299, 2005. 

• Biomedical and Digital Human related slides are generated using available info in public domain, such 
as in Journals and university colleagues (Prof. Tim David of Canterbuty University, NZ and, Prof. Stan 
Berger of UC Berkeley) 


Outline of Talk 


• Introduction / Background 

• Modeling and Simulation Challenges in Aerospace Engineering 

• Modeling and Simulation Challenges in Biomedical Engineering 
- Digital Astronaut 

• Project Columbia 


Summary and Discussion 


Status of CFD 


• Pioneered the field of flow simulation for 

- Obtaining engineering solutions for complex configurations (One of major goals of 80s) 

- Understanding flow physics (critical to mission) 

• Functionality was the most important features primarily concerned about 
"performance” prediction (of aerospace vehicles), and then reduce design cycle 
time 

- CFD is routinely used in aerodynamic design of aircraft 

=> There are a few remaining challenges, e.g. high-lift, aeroacoustics 

- Design of aircraft engine components is validated by CFD 
=> CFD-based engine design is yet to be realized 

- CFD can provide detailed analysis of rocket engine components 

=> Analysis/design of entire engine / entire sub-systems are yet to be accomplished 

• There are many success stories 

- Commercial aircraft design 

- SSME HGM redesign 

- Mechanical heart assist device 
.. list goes on 

• New advances are often driven by problems or mishaps 


CFD Contribution to Commercial Aircraft Design 
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Examples of Current Simulation Capabilities for Complex Flow 



Steady and unsteady flows over complex 
configurations 

Bodies in relative motion 
Multiple scale problems 
CFD very good at: 

- Flow analysis 

- Sorting through preliminary designs 
Still lacks 

- Prediction capability 

- Automation 


Historical Note 


« Space Shuttle Challenger Disaster has lead to key technological 
advances that facilitate computational aerodynamic forensics 











Historical Note 



Space Shuttle Main Engine Phase 11+ redesign work in the early 80s 
accelerated CFD applications to rocket propulsion systems development 

Propulsion CFD. Pump New flight Engine in 1995 

technology in the early 90s 


CFD solution of HGM (c 1983) 


Integrated Simulation Environment 


Analysis & Visualization 
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Grand Challenges 


Data & Computation 
Management 
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HEC Challenges in Aerospace Engineering 


Primary focus of “Modeling and Simulation” has been changed from 
performance prediction to mission impact 

• Expand flight envelope, reduce design cycle time, and enhance safety 

- Digital Flight 

6-DOF, S&C database generation, Grid quality issues etc. 

• Mission simulation 

- Integrated Launch/Ascent Simulation 
e.g. Shuttle’s Return To Flight 

- Descent / Entry / Landing 

- Mixed Fidelity Systems Simulation for Risk Evaluation 

• High-fidelity simulation of complex system 

- Liquid Rocket Subsystem Simulation 

e.g. turbopump systems: induce, impeller, diffuser etc. 

• Bioastronautics 

- Digital Astronaut 

=> Requires increased processor speed, larger memory, data analysis and 
management, faster network: CFD, CS and Facility 
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Integrated Launch/Ascend Simulation 


Goal: 

To provide automated high-fidelity simulation 
capability for vehicle in launch/ascent 
environment. 

Objectives: 

•Mission design & analysis 

•Plan & evaluate readiness to launch 

•Trajectory optimization 

•Risk and safety evaluation 

•Failure/abort recovery analysis 

•Rapid response to critical events 
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High-Fidelity Launch/Ascent Simulation Features 


Example of Overset Grid System and Solution 


• Rapidly generate grid systems for 
different control-surface deflections, and 
different combinations of components 

• 1 hour to change control-surface 
deflections • / 

• Overset grid generation utilizes Chimera 
Grid Tools and Pegasus5 software 


• Rapid turn around 

• 24 hours of wailclock time - 

• 32 SGI Altix CPUs (ltanium-2 CPUS) 


Launch Tower 
Multiple Bodies 


Thermal Stres 




Example of Current Moving Body Capability 


® Multipie-exposure of moving-body simulation of debris event 
s Dozens of other examples in literature 

Moo = 2.46 
a = 2 . 08 ° 



Isobars 

multiple exposure 


High-Fidelity Launch/Ascent Simulation - “Digital Launch” 


Modeling and Simulation Challenges 

• Complexity 

• 6-DOF multiple-body flight 

• Debris impact scenarios 

• Integrate vibration data and fuel exhaust data from propulsion simulation 

• Integrate acoustic environment and fuel accumulation from engine exhaust 

• Thermal stress analysis on structure 

Computation 

• Simulation until vehicle has passed tower (e.g. 3 sec.) 

• Time step determined by exhaust blast waves 

• Length scale assumed 1 foot 

• Time accurate Euler simulation for most of ascent 

• 100 million grid points 

• Current Solution Time: less than one day on 12TF Platform 

• Real Design & Engineering Mode! for Ascent/Abort Scenario: 

•Weather 
•Viscous effects 
•Exhaust Chemistry 

•Acoustics environment (ignition overpressure) 

HEC Requirement for High-Fidelity Design 

• 1 day & 60-120 TF Platform 



Liquid Rocket Subsystem Simulation 


Goal: 

To provide a high-fidelity computational 
framework for design and analysis of the 
fuel/oxidizer supply subsystem for liquid 
rocket propulsion systems. 


Objective: 

• Decrease design cost 

• Improve performance and reliability 

Critical path for: 

- Retrofitting (i.e. determining fuel-line 
crack issue for Space Shuttle Program) 

- Investigating new designs for future 
engines. 


EXAMPLE OF A ROCKET ENGINE PUMP 



Turbopump Design Problems 


Can lead to cost growth, engine failure or vehicle loss 



Cavitation induced 
environments 


Interaction 
between pump 
and feed line 
leads to flow- 
liner cracks 


Impeller unsteady loads 


Loads and 
interaction with 
preburner 


Turbine 
unsteady loads 


Liquid Rocket Fuel System Cause of Most Failures 




Wide-Range Flow Impeller-Diffuser 



Code INS3D 

34.3 Million Grid Points, 114 Zones 

One Rotation takes 3.5 days on 128 SGI Origin (600 MHz) CPU’s [.015 TF] 

Source: Kiris, et. al., NASA ARC 


Liquid Rocket Subsystem Simulation 


Modeling and Simulation Challenges: 

•Complex geometry: CAD-to-solution scripting capability. 

•Boundary layer transition and separation 
•Transient flow phenomena. 

•Tip Vortices 
•Cavitation 

•Parallel, scalable computational procedure. 

•Systems analysis capability for trade study in designing components 
•Accurate prediction of flow-induced vibration 
Computation 
•Simplified Model 

- upstream and downstream manifolds/ducting, and coupled shroud and hub cavities, ignores 
cavitation, 

- 150 million grid points 

- 10 revolutions 

- Extrapolate INS3D 

- Requires four days on 12 TF Platform 

HEC Requirement for High-Fidelity Design 

- Fuel system with piping and four turbopumps 

- 2 day solution & 240-500 TF Platform 


Human Modeling and Simulation 


Potential Applications 
Biomedical Research: 

• To interpret large quantities of data, guide experiments 

• Drug-design, development and testing 

Medical Practice: 

• Diagnosis- Identify irregularities 

• Treatment- Plan treatment procedures 

• Monitoring- during space flight 

Education and training: 

• Cell-level computation, organ modeling, gross anatomy etc. for K-12 to 
practitioners 

Modeling human factors, disease control etc... 

There are many players 

• NIH and their Affiliates 

• DoD 

• NASA - Bioastronautics (Digital Astronaut is an element in this) 

• Universities - Bioengineering Centers and Medical Schools 

• Industry 


Digital Astronaut 


Goal: 

To provide high-fidelity simulation for 
predicting human performance in space 
flight and develop countermeasures 


Objectives: 


Assess impact of altered gravity on crew 
during mission and post-mission recovery 


Guide counter meast3f;e development 



Approach: 

• Develop integrated DA system. 
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This work is being performed in collaboration 
with other biomediclfresearch organizations 


INTEGRATED IMPACT ON 
ASTRONAUT'S PERFORMANCE 
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Circulatory System Simulation for Human Space Flights 


Scenario: Blood circulation will have major impacts on crew during and post mission. 
Simulation of entire human circulatory system under space enviro nment will g uide long 
duration human space flights. 


Modeling and Simulation Challenges: 

• Digitize anatomical data - Mapping MRA data image §9 

processing 

• Computational hemodynamics - time dependant flow ( 

• Blood flow simulation in large and small arteries 

- Non-Newtonian modeling and gravitational effects 

• Biologically complex (cell chemistry) 

• Structural computation of vessel wall and diffusion of 
molecules through the wall 

• Auto-regulation and control model 

- Boundary conditions for truncated model 

• Difficult to validate, but possible 

• Multi-disciplinary team ' " ► : 

FLOW 

- Collaborators often geographically far apart 

MOLECULAR DYNAMICS 
IN CAPLILARIES 




BIFURCATION 


Circulation-Based Performance Model 
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‘High-fidelity model -for selected area - Brain 




Why brain/blood flow first? 


Biomedical Study 

• Stroke is the third largest cause of death 

• Stroke is the leading cause of long term disability 

• Incidence of stroke is projected to triple in the next 30 years 

Bioastronautics / Digital Astronaut 

• High G manoeuvres induce significant blood flow variations with ischaemic 
consequences. 

- Black-out: 

+8G causes unconciousness 

- Red-out: 

-3G makes retina engorged with blood 
Optimal blood supply to the brain is crucial for astronauts performance 


Brain Schematic 


• CCA (Common Carotid Arteries) 

• ECA (External Carotid Arteries) : 
Supply blood to face and scalp 
area 

• ICA (Internal Carotid Arteries): 

• Supply blood to the front 3/5 of 
the cerebrum 

• Vertebrobasilar Artery 

• Supply back 2/5 of the cerebrum, 
part of the cerebellum, and the 
brain stem 



Source- http://www. strokecemer.org 





Cerebrovascular Geometry 


/ he Circle of Willis (CoW) 

• Located at the Base of the Brain, 
beneath the hypothalamus 


• Responsible for collateral distributing 
blood to the major regions of the 
brain. 


The Anatomical Dilemma 



•Approximately 50% of the 

population has a complete CoW ^ XA 


•Variation in the communicating 
arteries is common. 



•Combined with the effects of 
hydrostatic pressure variation 
can cause cerebral dysfunction 



Anatomical Variations 
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Anatomical Variations 
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Variations 






Code Validation: Carotid bifurcation 





• CCA Diameter, 2r = 8 mm 

• Flow ratio, Qe/Qc = 0.45 

• Mean velocity, U = 7 cm/s 

• KSCN Density, rho = 1410 kg/m 3 

• KSCN Viscosity, v = 2.9 cPoise 
0.0029 Pa s 

• Reynolds number, Re = 270.0 

• (Exp data from Gijsen, et al. 1999a) 

• Chimera overset grid w/ 8 blocks 

• Grid size: 41x21x25=21,525 
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Code Validation: Carotid bifurcation 




Effects of Wall Distensibility 



Effects of Wall Distensibility 



Temporal wall shear stress during a heart pulse (Re=388) 


Effects of Wall Distensibility 



Axial velocity profiles due to gravitational variation (Re-388) 
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(a) Standing 


(b) Supine 


(c) Hand-Standing 


Wall shear stress distribution due to gravity variation 


Geometry Definition 


Image segmentation from MR Angiography : 3D Reconstruction 




Image Extracting 






Geometry Definition 



Magnetic Resonance Images (MRI) of human 

Source: Professor Tim David, University of Canterbury, New Zealand 


Coronal (Rear) 


Transverse (Top) 


Sagittal (Side) 


Grid generated by digitized xyz data of the MRI above 





Circle of Willis Simulation: Instantaneous Pressure 



•Flow conditions: 

•Left ICA diameter, 2r = 5.6 mm 
•Mean velocity, U = 14 cm/s 
•Blood density, rho = 1054 kg/m** 
3 

•Blood viscosity, v = 3.5 cPoise = 
0.0035 Pa s 

•Reynolds number, Re = 240.0 


Code: INS3D 
2 Million Grid Points 
1 day to complete on 128 p. 
SGI Origin 3000 (600Hz) 
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Biomedical Challenges 


• How can we derive models which “replicate” human geometry and provide 
viable (i.e. believable to a physiologist) data in a reasonable timescale ? 

• Investigate the coupling of complex cellular reaction mechanisms with 
mass transfer models and complex fluid dynamics (remember the 
geometry !) 

• How to include hundreds of cell mechanisms that function in a healthy 
state (i.e. Ca2+) 

• Fluid-structure interaction needs to be included 

*Venous and lymph valves 
^Artificial Heart Valves 

-250,000 replacements in the US per year 
-Mechanical -anticoagulant therapy 
-Flexible - no drugs needed but NOT DURABLE 


Circle of Willis Simulation 

Particles colored by velocity magnitude 



Some more math 


• Try this arithmetic exercise: 

How many times does your heart beat every year ? 

37 . 000 . 000 (100,800 more in a leap year!) 

And every decade ? 

370 . 000 . 000 (obviously !) 

In your life (Let’s say 80 years !) ? 

30 . 000 . 000.000 


More Challenges 


• What are the fundamental parameters governing the proper, long- 
term functioning of heart valves, and how do we test for durability ? 

• If we do accelerated testing are the dynamics different ? 

• Does the valve operate differently in micro-gravity ? 




Heart-Brain Circulation Dynamics 


Why Heart-Brain circulation model first 

• Further development of complex physioloj 
models of auto-regulation in micro gravity 

heart-to-head 

• Pathways of large interacting particles, 
embolic stroke 


Carotid Bifurcation (Wall shear) 


High-Fidelity Biomedical Computing 


Flows in Atherosclerotic Stenotic Vessels 

•Atherosclerosis causes gradual narrowing 
of arteries due to plaque growth 


•Sufficient narrowing, or plaque rupture 
can lead to flow blockage and resulting 
tissue ischemia, leading in heart to cardiac 
infarct, in brain to stroke. 


•Flow blockage and mechanical loading 
are important, for example, to determine 
tissue ischemia 


Computation 

Angiogram 


Experiment 


•Local flow is neither fully turbulent nor 
laminar ( chaotic . disturbed transitional 
etc ) 

=>HEC will shed light on this local 
phenomena thru fine resolution 
computation (LES/DNS equivalent) 


Detailed flow analysis, including arterial 
wall structural model and blood 
rheology, to explain major cause of 
cardiac infarct and/or stroke 





Integrated Digital Astronaut System 


Digital Astronaut Performance Prediction 
- To predict astronaut performance during short- 
and longer-duration space flight 
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Digital Astronaut 
Central Integrating 
System 

(Circulatory 
Dynamics-Based 
Modified Guyton 
Model) 


Image-Guided Monitoring 

-High fidelity simulations of crew anatomy and 

physiology to monitor flight risks and to provide 

countermeasures to mitigate 

•Effects of Radiation 

•Effects due to aravitational and environmental 


factors 



Liver and Intestine for 
drug/nutrient absorption 
evaluation 



3D reconstruction of 
ascending vasculature from 
the aortic arch to the Circle 
of Willis for hemodynamic 
modeling 



Kidney for monitoring renal 
function and fluid 
processing capabilities 



4D reconstruction of the 
heart for simulating 
changes to cardiac 
activity 


“Columbia” Supercomputer, 
Storage, & Networks enables 
high-fidelity Digital Astronaut 
System 



“Columbia”: World Class Supercomputing 


• The NAS houses the world’s fastest 
operational supercomputer 
providing 61TF of compute 
capability to the NASA user 
community 

• Columbia is a 20-node 
supercomputer built on proven 512- 
processor nodes 

• Columbia is the largest SGI system 
in the world with over 10,000 Intel 
Itanium2 processors 

• Columbia provides the largest node 
size incorporating commodity parts 
(512) and the largest shared 
memory environment (2048) 
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Circle of Willis Simulation 


Particles colored by velocity magnitude 


Biomedical Challenges 


How can we derive models which “replicate” human geometry and provide 
viable (i.e. believable to a physiologist) data in a reasonable timescale ? 

Investigate the coupling of complex cellular reaction mechanisms with 
mass transfer models and complex fluid dynamics (remember the 
geometry !) 

How to include hundreds of cell mechanisms that function in a healthy 
state (i.e. Ca2+) 

Fluid-structure interaction needs to be included 
*Venous and lymph valves 
^Artificial Heart Valves 

-250,000 replacements in the US per year 
-Mechanical -anticoagulant therapy 
-Flexible - no drugs needed but NOT DURABLE 





Some more math 


• Try this arithmetic exercise: 

How many times does your heart beat every year ? 

37 . 000 . 000 (100,800 more in a leap year!) 

And every decade ? 

370 . 000 . 000 (obviously !) 

In your life (Let’s say 80 years !) ? 

30 . 000 . 000.000 


More Challenges 


• What are the fundamental parameters governing the proper, long- 
term functioning of heart valves, and how do we test for durability ? 

• If we do accelerated testing are the dynamics different ? 

• Does the valve operate differently in micro-gravity ? 


High Fidelity Modeling and Simulation 

- More accurate : enhanced via more physics based simulation 

- Enabling : some mission characteristics not understood otherwise 

- Affordable and becoming more cost effective 

e.g. “Columbia” offers a quantum leap in HEC resources at NASA 

Resources and time 

- Will help sustain future mission 

(ground and flight testing are expensive and time consuming) 

Use of best simulation tools can 

- Increase reliability and reduce mission risk 


